[INFO] cloning repository https://github.com/xiaokai1996/llm_pretrain_data_dedup
[INFO] running `Command { std: "git" "-c" "credential.helper=" "-c" "credential.helper=/workspace/cargo-home/bin/git-credential-null" "clone" "--bare" "https://github.com/xiaokai1996/llm_pretrain_data_dedup" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fxiaokai1996%2Fllm_pretrain_data_dedup", kill_on_drop: false }`
[INFO] [stderr] Cloning into bare repository '/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fxiaokai1996%2Fllm_pretrain_data_dedup'...
[INFO] running `Command { std: "git" "rev-parse" "HEAD", kill_on_drop: false }`
[INFO] [stdout] 1d7c7e98b73fd86a254d43a7eccd8ea8c96b8542
[INFO] checking xiaokai1996/llm_pretrain_data_dedup against master#4eb62535fc12dd1a63fd43a4173e224e79313c4d for pr-158816
[INFO] running `Command { std: "git" "clone" "/workspace/cache/git-repos/https%3A%2F%2Fgithub.com%2Fxiaokai1996%2Fllm_pretrain_data_dedup" "/workspace/builds/worker-4-tc1/source", kill_on_drop: false }`
[INFO] [stderr] Cloning into '/workspace/builds/worker-4-tc1/source'...
[INFO] [stderr] done.
[INFO] started tweaking git repo https://github.com/xiaokai1996/llm_pretrain_data_dedup
[INFO] finished tweaking git repo https://github.com/xiaokai1996/llm_pretrain_data_dedup
[INFO] tweaked toml for git repo https://github.com/xiaokai1996/llm_pretrain_data_dedup written to /workspace/builds/worker-4-tc1/source/Cargo.toml
[INFO] validating manifest of git repo https://github.com/xiaokai1996/llm_pretrain_data_dedup on toolchain 4eb62535fc12dd1a63fd43a4173e224e79313c4d
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+4eb62535fc12dd1a63fd43a4173e224e79313c4d" "metadata" "--manifest-path" "Cargo.toml" "--no-deps", kill_on_drop: false }`
[INFO] crate git repo https://github.com/xiaokai1996/llm_pretrain_data_dedup already has a lockfile, it will not be regenerated
[INFO] running `Command { std: CARGO_HOME="/workspace/cargo-home" RUSTUP_HOME="/workspace/rustup-home" "/workspace/cargo-home/bin/cargo" "+4eb62535fc12dd1a63fd43a4173e224e79313c4d" "fetch" "--manifest-path" "Cargo.toml", kill_on_drop: false }`
[INFO] [stderr] warning: Cargo.toml: `package.edition` is unspecified, defaulting to `2015` while the latest is `2024`
[INFO] [stderr] warning: `dedup_dataset` (manifest) generated 1 warning
[INFO] [stderr]     Updating crates.io index
[INFO] [stderr]  Downloading crates ...
[INFO] [stderr]   Downloaded filebuffer v0.4.0
[INFO] [stderr]   Downloaded zstd-safe v2.0.6+zstd.1.4.7
[INFO] [stderr]   Downloaded zstd v0.5.4+zstd.1.4.7
[INFO] [stderr]   Downloaded crossbeam v0.3.2
[INFO] [stderr]   Downloaded zstd-sys v1.4.18+zstd.1.4.7
[INFO] running `Command { std: "docker" "create" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/source:/opt/rustwide/workdir:ro,Z" "-v" "/var/lib/crater-agent-workspace/builds/worker-4-tc1/target:/opt/rustwide/target:rw,Z" "-v" "/var/lib/crater-agent-workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/var/lib/crater-agent-workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-m" "1610612736" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:3d5ced03c013a94a2f102a4510f48a6e9184255caf5fd8244f58017bde7f5210" "sleep" "infinity", kill_on_drop: false }`
[INFO] [stdout] 08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016
[INFO] running `Command { std: "docker" "start" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-w" "/opt/rustwide/workdir" "--user" "0:0" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016" "/opt/rustwide/cargo-home/bin/cargo" "+4eb62535fc12dd1a63fd43a4173e224e79313c4d" "metadata" "--no-deps" "--format-version=1", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "inspect" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "CARGO_INCREMENTAL=0" "-e" "RUST_BACKTRACE=full" "-e" "RUSTFLAGS=--cap-lints=forbid" "-e" "RUSTDOCFLAGS=--cap-lints=forbid" "-w" "/opt/rustwide/workdir" "--user" "0:0" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016" "/opt/rustwide/cargo-home/bin/cargo" "+4eb62535fc12dd1a63fd43a4173e224e79313c4d" "check" "--frozen" "--all" "--all-targets" "--message-format=json", kill_on_drop: false }`
[INFO] [stderr] warning: Cargo.toml: `package.edition` is unspecified, defaulting to `2015` while the latest is `2024`
[INFO] [stderr] warning: `dedup_dataset` (manifest) generated 1 warning
[INFO] [stderr]    Compiling libc v0.2.181
[INFO] [stderr]    Compiling version_check v0.9.5
[INFO] [stderr]    Compiling proc-macro2 v1.0.106
[INFO] [stderr]    Compiling unicode-ident v1.0.23
[INFO] [stderr]    Compiling either v1.15.0
[INFO] [stderr]    Compiling shlex v1.3.0
[INFO] [stderr]    Compiling find-msvc-tools v0.1.9
[INFO] [stderr]    Compiling quote v1.0.44
[INFO] [stderr]    Compiling glob v0.3.3
[INFO] [stderr]    Compiling syn v1.0.109
[INFO] [stderr]    Compiling autocfg v1.5.0
[INFO] [stderr]    Compiling radium v0.7.0
[INFO] [stderr]    Compiling zstd-safe v2.0.6+zstd.1.4.7
[INFO] [stderr]     Checking os_str_bytes v6.6.1
[INFO] [stderr]     Checking tap v1.0.1
[INFO] [stderr]     Checking hashbrown v0.12.3
[INFO] [stderr]    Compiling heck v0.4.1
[INFO] [stderr]     Checking wyz v0.5.1
[INFO] [stderr]     Checking bitflags v1.3.2
[INFO] [stderr]     Checking funty v2.0.0
[INFO] [stderr]     Checking once_cell v1.21.3
[INFO] [stderr]     Checking strsim v0.10.0
[INFO] [stderr]     Checking textwrap v0.16.2
[INFO] [stderr]     Checking clap_lex v0.2.4
[INFO] [stderr]     Checking termcolor v1.4.1
[INFO] [stderr]    Compiling itertools v0.9.0
[INFO] [stderr]    Compiling proc-macro-error-attr v1.0.4
[INFO] [stderr]    Compiling proc-macro-error v1.0.4
[INFO] [stderr]     Checking crossbeam v0.3.2
[INFO] [stderr]    Compiling indexmap v1.9.3
[INFO] [stderr]     Checking bitvec v1.0.1
[INFO] [stderr]     Checking atty v0.2.14
[INFO] [stderr]     Checking filebuffer v0.4.0
[INFO] [stderr]    Compiling jobserver v0.1.34
[INFO] [stderr]    Compiling cc v1.2.55
[INFO] [stderr]    Compiling zstd-sys v1.4.18+zstd.1.4.7
[INFO] [stderr]    Compiling clap_derive v3.2.25
[INFO] [stderr]     Checking clap v3.2.25
[INFO] [stderr]     Checking zstd v0.5.4+zstd.1.4.7
[INFO] [stderr]     Checking dedup_dataset v1.0.0 (/opt/rustwide/workdir)
[INFO] [stdout] warning: unused variable: `thread_sum`
[INFO] [stdout]    --> src/main.rs:798:10
[INFO] [stdout]     |
[INFO] [stdout] 798 |         let thread_sum:usize = result.into_iter().map(|t| t.join()).sum();
[INFO] [stdout]     |             ^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_thread_sum`
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: unused variable: `thread_sum`
[INFO] [stdout]    --> src/main.rs:798:10
[INFO] [stdout]     |
[INFO] [stdout] 798 |         let thread_sum:usize = result.into_iter().map(|t| t.join()).sum();
[INFO] [stdout]     |             ^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_thread_sum`
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `is_present_memory` is never used
[INFO] [stdout]    --> src/main.rs:442:4
[INFO] [stdout]     |
[INFO] [stdout] 442 | fn is_present_memory(text: &[u8],
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: call to `.clone()` on a reference in this situation does nothing
[INFO] [stdout]     --> src/main.rs:1447:84
[INFO] [stdout]      |
[INFO] [stdout] 1447 | ...   if !path.starts_with(&Path::new(cache_dir).join(format!("dups_{}_", ds_name.clone())).into_os_string().into_string().unwrap(...
[INFO] [stdout]      |                                                                                  ^^^^^^^^ help: remove this redundant call
[INFO] [stdout]      |
[INFO] [stdout]      = note: the type `str` does not implement `Clone`, so calling `clone` on `&str` copies the reference, which does not do anything and can be removed
[INFO] [stdout]      = note: `#[warn(noop_method_call)]` on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: function `is_present_memory` is never used
[INFO] [stdout]    --> src/main.rs:442:4
[INFO] [stdout]     |
[INFO] [stdout] 442 | fn is_present_memory(text: &[u8],
[INFO] [stdout]     |    ^^^^^^^^^^^^^^^^^
[INFO] [stdout]     |
[INFO] [stdout]     = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stdout] warning: call to `.clone()` on a reference in this situation does nothing
[INFO] [stdout]     --> src/main.rs:1447:84
[INFO] [stdout]      |
[INFO] [stdout] 1447 | ...   if !path.starts_with(&Path::new(cache_dir).join(format!("dups_{}_", ds_name.clone())).into_os_string().into_string().unwrap(...
[INFO] [stdout]      |                                                                                  ^^^^^^^^ help: remove this redundant call
[INFO] [stdout]      |
[INFO] [stdout]      = note: the type `str` does not implement `Clone`, so calling `clone` on `&str` copies the reference, which does not do anything and can be removed
[INFO] [stdout]      = note: `#[warn(noop_method_call)]` on by default
[INFO] [stdout] 
[INFO] [stdout] 
[INFO] [stderr]     Finished `dev` profile [optimized + debuginfo] target(s) in 1m 00s
[INFO] running `Command { std: "docker" "inspect" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016", kill_on_drop: false }`
[INFO] [stdout] 08f5b3f61f357412f74f2e51b5a8f81a2f614ca4c70d69f120f0eccd74df9016
